node-api: widen the measured slice, harden the harness, record the runs to 72.3% - #6
Open
Alan Wisper (wisper-dev) wants to merge 8 commits into
Open
node-api: widen the measured slice, harden the harness, record the runs to 72.3%#6Alan Wisper (wisper-dev) wants to merge 8 commits into
Alan Wisper (wisper-dev) wants to merge 8 commits into
Conversation
Node's core suite spawns children that outlive the test process and hold its stdout open, so killing the entry process alone left the read hanging. The process now runs in its own group and a timeout reaps the tree; a descendant that opened its own session escapes the kill, so the pipes are dropped after a two-second grace.
`--elide` usually points at the `dist/current` symlink, and the image build context must carry the real files; the staged copy now dereferences instead of building around a broken link.
The workload sets its own case timeout; the test asserted the previous default.
Adds the missing subsystem groups (child_process, cluster, dgram, domain, http, net, os, readline, tty, zlib and more) so the suite measures the public Node surface rather than a sparse slice. Each test now runs with a heap bound: the suite contains cases whose only brake is stream backpressure (test-zlib-brotli-16GB.js decompresses 16 GB), and unbounded such a case exhausts the host instead of failing. A static expectations file carries the skips with a reason each: the cluster group is out of project scope for now, and the node:vfs tests cover an experimental Node subsystem Elide does not implement.
Records the measured progression on the widened slice: 35.8%, 43.2%, 50.4%, 60.1% and 65.0% (1,387 of 2,133 counting the denominator without skips). The latest run carries its full report body; the earlier ones keep the raw results, counts and change summary, which is what the trend, the run database and the next run's diff read. The ratchet is retaken against the latest run and holds 857 known failures.
Replaces the July handoff doc: the failure map by subsystem with the hang subset broken out, what the static expectations skip and why, the ranked work items that follow, and run instructions that match the current selection.
Node's suite contains tests that fork without bound, and the timeout-driven process-group kill only fires once a runaway has already taken the machine's memory. Each test now runs in a transient systemd scope with memory and task ceilings plus a runtime cap that reaps the whole cgroup, descendants that called setsid included; the harness container gets its own memory and pid limits with swap disabled. Where cgroups are unavailable — inside the container, on macOS — the run proceeds as before.
Probing the cgroup cage no longer fails the run: `Bun.spawnSync` throws for a missing executable rather than reporting an exit code, so inside the harness container — where `systemd-run` does not exist — the whole suite died before its first test. An absent `systemd-run` now means "run uncaged", as intended, and the container's own limits apply. A test covers that branch: it strips the PATH of the child it spawns rather than mutating `process.env`, because the search path a process uses is the one it started with. The run itself, on Elide 1.4.4+405f52319: 1,542 of 2,133 excluding skips — 72.3%, up from 65.0%, with 577 new passes and 194 fixed. It is RED: 18 tests the ratchet expects to pass do not, 7 of them timeouts clustered in http and net, and 11 functional, led by `test-stream-pipeline-duplex` and `test-stream-pipe-flow-after-unpipe`. The ratchet is deliberately left as it was, so those 18 stay visible rather than being absorbed into the baseline. The briefing is rewritten around the new numbers: the failure map (591 files, http now the largest block at 79 with 26 hangs), the largest root-cause signatures, and a work order that puts the regressions first.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
node-api: widen the measured slice, harden the harness, record the runsBrings the
node-apisuite to an honest measurement and records the progression35.8% → 43.2% → 50.4% → 60.1% → 65.0% → 72.3%. The latest run is 1,542 passes
out of 2,133 counting the denominator without skips.
That run is RED: 18 tests the ratchet expects to pass do not — 7 timeouts,
clustered in http and net, and 11 functional failures. The ratchet is
deliberately left as it was, so those 18 stay visible instead of being absorbed
into the baseline.
Harness
that outlive the test process and hold its stdout open, so killing the entry
process alone left the output read hanging and the run stalled. The process now
gets its own group and a timeout reaps the tree. A descendant that opened its
own session still escapes the group kill, so the pipes are dropped after a
two-second grace.
--elideusuallypoints at the
dist/currentsymlink, and the image build context must carrythe real files; the staged copy now dereferences instead of building around a
broken link.
systemd-runno longer fails the run:Bun.spawnSyncthrows for a missing executable rather than reporting an exitcode, so inside the harness container the whole suite died before its first
test. An absent
systemd-runnow means "run uncaged" and the container's ownlimits apply. A test covers that branch: it strips the PATH of the child it
spawns rather than mutating
process.env, because the search path a processuses is the one it started with.
the previous default.
Test selection
child_process,cluster,dgram,domain,http,https,net,os,perf_hooks,querystring,readline,tty,v8,zliband more, so the suite measures the public Nodesurface rather than a sparse slice.
-XX:MaxHeapSize=4g. The suite contains cases whoseonly brake is stream backpressure (
test-zlib-brotli-16GB.jsdecompresses16 GB); unbounded, such a case exhausts the host instead of failing.
expectations/node-api.tomlcarries the skips witha reason each: the
clustergroup is out of project scope for now (one glob, sothe decision stays visible and reversible), and the
node:vfstests cover anexperimental Node subsystem Elide does not implement.
Ratchet and reports
the newest run, so the 18 regressions above stay reported rather than absorbed.
map is the working document for the next batch of fixes. The earlier ones
keep only the raw results, counts and change summary: that is what the trend,
the run database (
ingestAll) and the next run's diff actually read, while therendered bodies are superseded as soon as a newer run lands.
BRIEFING-node-api.mdis rewritten around the current measurement: the failuremap by subsystem with the hang subset broken out, what the static expectations
skip and why, the ranked work items that follow, and run instructions matching
the current selection.
Checks
bun --cwd harness test— 172/172 pass.bun run typecheck— clean.